test: drain a subnet that is "cooling down" - #11371

Adds a `merge_subnets` endpoint to the registry canister. It takes a source
and a destination subnet ID and merges the canister ID ranges of the source
subnet into the canister ID range set of the destination subnet, so that the
canisters hosted by the source subnet are routed to the destination subnet
afterwards.

The routing table is updated via `modify_routing_table`, which assigns the
source subnet's ranges to the destination subnet and coalesces the resulting
adjacent ranges. Only the routing table is touched: neither subnet record is
modified and the source subnet is not deleted, so decommissioning a subnet is
`merge_subnets` followed by `delete_subnet`.

The endpoint is restricted to governance and rejects payloads where the two
subnet IDs are equal, either subnet is unknown, the source subnet hosts no
canister ID range, or an ongoing canister migration overlaps the source
subnet's ranges (which would leave those migrations with a host that is not on
their recorded trace).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A system test covering the `cooling_down` subnet record flag end to end: while
two universal canisters keep calling each other across subnets in a loop,
labeling one of their subnets as "cooling down" must bring that subnet to a
standstill.

The IC consists of an NNS subnet (with the NNS canisters installed) and two
Application subnets S and T, holding one universal canister each. Both
canisters are made to call the one on the other subnet in a loop, where the
reply (or reject) callback of every call fires a new call. A payload cannot
contain itself, so each canister holds the loop body in its global data and the
continuation calls the canister itself, passing the global data as the payload
to execute; the loop body replies as soon as it has fired the cross-subnet
call, which keeps the number of open call contexts bounded.

An `UpdateConfigOfSubnet` proposal then labels S as "cooling down" and the test
waits until S rejects ingress messages and, subsequently, until it is "merge
ready" according to the `Merge readiness` panel of the `Subnet merging`
dashboard (`bases/apps/ic-dashboards/core/subnet-merging.json` in
dfinity-ops/k8s) for `V` = the registry version created by the proposal and
`R` = 0 cycles: every subnet has reached V, no stream in either direction holds
a message (loopback included), the ingress history holds nothing but
`processing` entries, S's subnet input and output queues are empty, its subnet
call context manager holds no call context, and the pending anonymous refunds
are worth at most R. As in the dashboard, each term is evaluated on the median
across the replicas reporting it, and missing data reads as zero. Finally, the
test checks that both loops are stalled, so that readiness is the consequence
of S cooling down rather than of the loops having stopped.

Only the HEAD NNS variant runs: the mainnet NNS canisters do not know the
`cooling_down` field of `UpdateSubnetPayload`, so Candid would silently drop it
and the proposal would be a no-op.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…wn-test'

…bnet-cooling-down-test'

…merge

Add the `MergeSubnets` NNS function, mapped to the `merge_subnets` method of the
registry canister under the `SubnetManagement` topic, and extend that method so
that a single such proposal performs the whole registry-side part of a subnet
merge, all in one registry version, so that the destination subnet never
observes a state where only some of it took effect:

  1. the canister ID ranges of the source subnet are merged into the canister ID
     range set of the destination subnet, so that all canisters that used to be
     hosted by the source subnet are routed to the destination subnet;
  2. a recovery catch-up package is created for the destination subnet, at the
     height, time and state hash of the merged state, running a fresh DKG for
     the destination subnet's membership; and
  3. the destination subnet is brought back online.

The payload therefore grows the `height`, `time_ns` and `state_hash` of the
merged state, plus an `initial_dkg_subnet_id` naming the subnet that handles the
`setup_initial_dkg` call: it must not be the destination subnet, which is
offline while the merge is in progress.

The caller is expected to have taken both subnets offline and to have extended
the state of the destination subnet with the state of the canisters of the source
subnet beforehand; `state_hash` is the hash of the manifest of the result. The
source subnet record is not modified and the source subnet is not deleted: it
merely does not host any canister ID range anymore.

The recovery catch-up package needs fresh DKG transcripts because its
`initial_ni_dkg_transcript_{low,high}_threshold` become the current transcripts
of the DKG summary that bootstraps consensus at the recovery height. The registry
only holds the destination subnet's genesis (or last recovery) catch-up package,
whose transcripts its nodes may no longer have the secret key shares for, so
reusing those would risk restarting a subnet that cannot sign.

Merging into a subnet holding chain keys is rejected: recovering such a subnet
requires resharing its keys onto the recovery catch-up package, which this
method does not do, and silently leaving the destination subnet unable to sign
would be worse than refusing.

As the method now makes an inter-canister call half way through, it follows
`do_recover_subnet` in checking that none of the records it is about to
overwrite -- and none of the canister ID ranges it validated -- changed while
that call was in flight. Unlike `do_recover_subnet`, it reports the reject code
and message if that call fails, rather than an opaque `unwrap` panic.

`StateMachine` only answered `setup_initial_dkg` requests from
`do_execute_round`, which `tick()` (and hence `await_ingress`) does not go
through, so any canister awaiting such a call hung there forever. Factor the
fake responses out into `setup_initial_dkg_responses` and produce them from
`tick_with_config` as well, next to the threshold signing requests it already
answers. Without this the integration test of the success path cannot run, which
is why it is part of this commit.

The two success unit tests of `merge_subnets`, which can no longer drive the
method to completion, now exercise the routing table part directly; the success
path as a whole, including the recovery catch-up package and the unhalting, is
covered by the integration test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Add a `checkpoint_time` subcommand printing the `batch_time_nanos` of the system
metadata of a checkpoint, i.e. the IC time the subnet had reached when it wrote
that checkpoint.

Merging a subnet into another one has to pick the block time the destination
subnet resumes from, which must be larger than the times of the checkpoints at
which both subnets were taken offline. Nothing printed a checkpoint's time so
far, so there was no way to check that from outside the replica.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…o batches

A subnet halting because of the `halt_at_cup_height` flag of its subnet record
logged nothing observable: the only trace it left was a `debug!`, which the
nodes' log level drops, so nothing outside the replica could tell that such a
subnet had come to a stop. The `is_halted` flag, in contrast, is reported at
info level by `ConsensusImpl::on_state_change`.

Raise that message to info level, so that both ways of halting a subnet can be
observed the same way. It is already emitted at most once every five seconds,
so this does not make the logs noticeably busier, and it names the height of the
first batch that is not delivered, i.e. one past the state the subnet stopped in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Extend the `cooling_down` system test past the point where the subnet is "merge
ready": it now goes on to merge that subnet away and to delete it, which is what
the cooling down and the merge readiness condition exist for. The destination is
a third Application subnet `R`; the NNS subnet stays available throughout, as it
is where the proposals of the test are executed, including the one recovering
`R` at the merged state.

The merge itself, once both subnets are "merge ready":

  - Both subnets are told to halt at their next CUP and are waited for. The wait
    reads each node's journal, where a halted replica reports that it delivers
    no batches, rather than watching for the height of the latest checkpoint to
    stop advancing: checkpoints are written once per DKG interval, i.e. minutes
    apart, so that height looks stable long before the subnet halts. Its
    checkpoint is then hundreds of rounds behind the state the merge readiness
    was established on, and holds, for instance, an `install_code` aborted at
    that checkpoint and only completed afterwards -- whose call context and
    reserved response slot the merge then drops, so that the destination subnet
    hits `[EXC-BUG] Could not find any install code call ...` and panics on
    resuming it.
  - The state of the merge is assembled from the two checkpoints the subnets
    halted at, as a new checkpoint of `R` at the next multiple of its DKG
    interval, so that `R`'s own checkpoint stays untouched: the canisters and
    canister snapshots of `M` are added to those of `R`, and the result is marked
    as the product of a subnet merge. Taking the system metadata and subnet
    queues of `R` wholesale is sound because those of `M` are empty, which is
    what the merge readiness established and what `M` cooling down preserves
    until it halts.
  - The block time of the merged state is picked past both checkpoint times,
    read with `state-tool checkpoint_time`, and its manifest hash with
    `state-tool manifest`.
  - The merged state is added to `R`'s checkpoints while its replica is stopped,
    and the replica is only started after the `MergeSubnets` proposal created the
    recovery CUP; the test then waits for `R` to report exactly that CUP, so that
    a subnet resuming from the wrong state fails here rather than much later.
  - The three endless loops are ended and every ingress message that was in
    progress across the merge is checked to have completed. This is what the
    `subnet_merged` marker is for: the ingress history of `M` is deliberately not
    merged in, the marker makes the replica re-register those messages instead.
  - Finally, once every subnet observed the merge, `M` -- which hosts no canister
    ID range anymore -- is deleted, and is checked to be gone from the registry.

`UniversalCanister::submit_update` now returns the ID of the message it
submitted, which is what lets the test track the ingress messages that outlive
the merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

# Conflicts:
#	rs/registry/canister/unreleased_changelog.md

Strengthen the subnet merge system test along the axes it was blind to.

Every subnet now has four nodes rather than one, so that the merge has to get the
merged state to the other nodes of the destination subnet the way a recovery
does, i.e. by state sync from the one node it is uploaded to, and so that the
medians the merge readiness condition is made of are medians of more than one
value.

The destination subnet is no longer empty: it hosts `UR`, which the merge has to
leave alone even though the canisters of the merged subnet are added right next
to it, and which has to be able to call one of those canisters afterwards, both
of them being on the same subnet from then on.

`U8` carries the state that has to survive the merge: a blob in its stable
memory, a canister snapshot, and a cycles balance, all three of which are checked
once the destination subnet serves it. The balance is compared against a share of
itself rather than an amount, as the two readings are however many minutes apart
the waits in between take, and what this catches is a balance that was not
carried over at all rather than the resource charges of an idle canister.

The merge readiness condition is now checked to *not* hold right after the subnet
starts cooling down. A condition that held from the start would be satisfied by a
subnet that never had anything to drain, so waiting for it would prove nothing;
six of its eight terms are in fact unsatisfied at that point.

Finally, a best effort call from `U9` on `T` to `U10` on `M`, which `U10` never
answers, puts a best effort message in flight across the merge: its deadline
passes while `M` is cooling down, and the cycles it carries are held by the open
call context of its callee, which the merge has to carry over like any other
canister state. The test had only guaranteed response calls until now.

The one term of the readiness condition that stays untested is the pending
anonymous refunds, which requires a cycle bearing message to be dropped from one
of the subnet's queues while owed to a canister of another subnet; the cycles of
the best effort call above are not it, as that call is picked up rather than
dropped. See the comment on `MAX_REFUND_VALUE_CYCLES`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…merge

Add the `MergeSubnets` NNS function, mapped to the `merge_subnets` method of the
registry canister under the `SubnetManagement` topic, and extend that method so
that a single such proposal performs the whole registry-side part of a subnet
merge, all in one registry version, so that the destination subnet never
observes a state where only some of it took effect:

  1. the canister ID ranges of the source subnet are merged into the canister ID
     range set of the destination subnet, so that all canisters that used to be
     hosted by the source subnet are routed to the destination subnet;
  2. a recovery catch-up package is created for the destination subnet, at the
     height, time and state hash of the merged state, running a fresh DKG for
     the destination subnet's membership; and
  3. the destination subnet is brought back online.

The payload therefore grows the `height`, `time_ns` and `state_hash` of the
merged state, plus an `initial_dkg_subnet_id` naming the subnet that handles the
`setup_initial_dkg` call: it must not be the destination subnet, which is
offline while the merge is in progress.

The caller is expected to have taken both subnets offline and to have extended
the state of the destination subnet with the state of the canisters of the source
subnet beforehand; `state_hash` is the hash of the manifest of the result. The
source subnet record is not modified and the source subnet is not deleted: it
merely does not host any canister ID range anymore.

The recovery catch-up package needs fresh DKG transcripts because its
`initial_ni_dkg_transcript_{low,high}_threshold` become the current transcripts
of the DKG summary that bootstraps consensus at the recovery height. The registry
only holds the destination subnet's genesis (or last recovery) catch-up package,
whose transcripts its nodes may no longer have the secret key shares for, so
reusing those would risk restarting a subnet that cannot sign.

Merging into a subnet holding chain keys is rejected: recovering such a subnet
requires resharing its keys onto the recovery catch-up package, which this
method does not do, and silently leaving the destination subnet unable to sign
would be worse than refusing.

As the method now makes an inter-canister call half way through, it follows
`do_recover_subnet` in checking that none of the records it is about to
overwrite -- and none of the canister ID ranges it validated -- changed while
that call was in flight. Unlike `do_recover_subnet`, it reports the reject code
and message if that call fails, rather than an opaque `unwrap` panic.

`StateMachine` only answered `setup_initial_dkg` requests from
`do_execute_round`, which `tick()` (and hence `await_ingress`) does not go
through, so any canister awaiting such a call hung there forever. Factor the
fake responses out into `setup_initial_dkg_responses` and produce them from
`tick_with_config` as well, next to the threshold signing requests it already
answers. Without this the integration test of the success path cannot run, which
is why it is part of this commit.

The two success unit tests of `merge_subnets`, which can no longer drive the
method to completion, now exercise the routing table part directly; the success
path as a whole, including the recovery catch-up package and the unhalting, is
covered by the integration test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

# Conflicts:
#	rs/registry/canister/unreleased_changelog.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Revalidate, after `setup_initial_dkg` returns, that no canister migration
overlapping the canister ID ranges of the source subnet was prepared while
the call was in flight: `prepare_canister_migration` does not touch the
routing table, so the existing post-call check could not catch it.

Also fix the low/high threshold transcript records being swapped in the
`SetupInitialDKGResponse` that `StateMachine` synthesizes for pending
`setup_initial_dkg` requests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Chain key initializations in a CUP take precedence over the chain key
configuration of the subnet record, so carrying over the ones of the CUP
being replaced would make the destination subnet bootstrap obsolete key
material -- contradicting the endpoint's refusal to merge into a subnet
holding chain keys. Clear both initialization fields, as recovering a
subnet without an initial chain key configuration does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Revert `merge_subnets` to what it originally did: merge the canister ID
ranges of the source subnet into the canister ID range set of the
destination subnet, and nothing else. The recovery catch-up package, the
fresh DKG for the destination subnet's membership and the unhalting are
dropped again, and with them the `height`, `time_ns`, `state_hash` and
`initial_dkg_subnet_id` payload fields, the inter-canister call and the
guards around it. Recovering the destination subnet at the merged state
stays with `recover_subnet`.

The `MergeSubnets` NNS function is kept -- without it the endpoint would
be unreachable -- with its description narrowed accordingly.

The `StateMachine` change that answered `setup_initial_dkg` requests from
`tick_with_config` goes away with the inter-canister call that needed it.
Its fix of the swapped `SetupInitialDKGResponse` fields stays, as that is
a bug of its own: the `HighThreshold`-tagged transcript was encoded as
the low-threshold record and vice versa.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…wn-test'

`merge_subnets` is restricted to the routing table change again, so the
registry side of the merge is taken from the merged-in branch wholesale:
the endpoint, its payload, the Candid interfaces, the changelogs and the
`StateMachine` handling of `setup_initial_dkg` requests. The extraction of
`setup_initial_dkg_responses` goes away with the inter-canister call that
needed it; the fix of the swapped `SetupInitialDKGResponse` fields stays.

The system test now performs the merge with two proposals instead of one:
`MergeSubnets` reroutes the canister ID ranges of `M` to `R`, and
`RecoverSubnet` creates the recovery CUP for `R` at the merged state.
Recovering a subnet that was instructed to halt at its next CUP replaces
that instruction with a plain halt, so `R` is unhalted with an
`UpdateConfigOfSubnet` proposal once it came up on the recovery CUP --
before the ingress messages that outlived the merge are checked to
complete, as a halted subnet delivers no batches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ing-down-test

…time

`await_status_is_healthy` and the SSH helpers without the `_async` suffix run
their own future through `futures::executor::block_on`, which polls it on the
calling thread instead of letting the runtime park on its I/O. A `reqwest`
request that finds no live connection in its pool then busy-polls its connect
future forever: the test spun at 100% CPU in `status_is_healthy_async`, having
logged the unhalting of `R` and nothing after it.

`R` reports `WaitingForRootDelegation` for a few minutes after the recovery,
which is what gave the pooled connection time to go away, so the health check
of step 16 hit this reliably. Use the `_async` variants -- which is what this
file already does for the checkpoint listing of `latest_checkpoint_height` --
for every driver call in the async body of the test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ing-down-test

# Conflicts:
#	rs/nns/governance/api/src/types.rs
#	rs/nns/governance/proto/ic_nns_governance/pb/v1/governance.proto
#	rs/nns/governance/src/gen/ic_nns_governance.pb.v1.rs
#	rs/nns/governance/src/proposals/execute_nns_function.rs
#	rs/nns/governance/unreleased_changelog.md
#	rs/registry/canister/src/mutations/merge_subnets.rs
#	rs/registry/canister/tests/merge_subnets.rs
#	rs/registry/canister/unreleased_changelog.md

The state side of a subnet merge was a sequence of `chmod`, `cp -al` and
`std::fs::write` calls in the subnet merge system test, which hand-encoded the
`subnet_merged` marker as the bytes `[0x08, 0x01]`. Move it into a `merge`
command of `state-tool`, next to the `split` command that performs the state
side of a subnet split.

The command assembles the checkpoint at `--output` from the ones at `--base`
(the destination subnet) and `--source` (the subnet being merged away): the
result holds everything of `base`, with the canisters and canister snapshots of
`source` added to those of `base`, and is marked as the product of a subnet
merge. Only canisters and snapshots are taken over from `source`; the ingress
history in particular is not, as the marker makes the replica re-register the
ingress messages of the merged-in canisters that are still in progress.

File contents are hard linked rather than copied, so this is cheap no matter how
large the two states are, and sound because checkpoints are immutable. Unlike
the shell version, the inputs may stay read-only: the linking creates the
destination directories itself rather than inheriting their permissions from a
`cp -al`.

The marker is written through `CheckpointLayout::subnet_merged_marker()` as a
`SubnetMerged` protobuf rather than by hand, and colliding canister IDs between
the two checkpoints are refused rather than silently resolved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…own-test

The state side of the subnet merge moved into a `merge` command of `state-tool`,
so drop the `chmod`/`cp -al`/`std::fs::write` sequence that did it here, along
with the `run_local` helper it was the only user of.

Adds a `merge` command to `state-tool`, performing the state side of a subnet
merge: the counterpart of the existing `split` command, which performs the state
side of a subnet split.

```
state-tool merge --base <destination checkpoint> \
                 --source <source checkpoint> \
                 --output <merged checkpoint>
```

The command assembles the checkpoint at `--output` from the ones at `--base`
(the subnet the canisters are merged into) and `--source` (the subnet being
merged away): the result holds everything of `base`, with the canisters and
canister snapshots of `source` added to those of `base`, and is marked as the
product of a subnet merge.

Only the canisters and their snapshots are taken over from `source`. Everything
else — system metadata, subnet queues, ingress history — is `base`'s. The
ingress history in particular is deliberately not merged in: the marker is what
makes the replica re-register the ingress messages of the merged-in canisters
that are still in progress.

File contents are hard linked rather than copied, so the assembly is cheap no
matter how large the two states are, and sound because checkpoints are
immutable: the links are only ever read afterwards. The two input checkpoints
may stay read-only, as the linking creates the destination directories itself
rather than inheriting their permissions from the tree it copies.

The marker is written through `CheckpointLayout::subnet_merged_marker()` as a
`SubnetMerged` protobuf rather than by hand, and canister IDs that collide
between the two checkpoints are refused rather than silently resolved: a
collision means the two checkpoints do not belong to the same merge.

The change is purely additive — a new command module plus its registration and
CLI wiring. No existing behaviour changes, and there are no new dependencies:
`ic-protobuf`, `ic-state-layout` and `ic-types` are already dependencies of
`state-tool`.

The rest of a subnet merge — halting both subnets, downloading the two states,
computing the batch time the merged state starts from, and proposing and
uploading it — is deliberately out of scope here. A subnet merge system test,
not yet on master, drives that workflow and calls this command for the assembly
step.

That test passes with the assembly performed by this command: the destination
subnet adopts the recovery CUP at the merged state, which means the manifest of
the assembled checkpoint matches the hash the recovery proposal was made with.
The test then checks one canister that the source subnet hosted, now served by
the destination subnet: it still holds the blob it had in its stable memory and
its canister snapshot, and its cycles balance dropped by no more than an idle
canister is expected to burn.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Assembling the merged state and adding it to the destination node were one step
that ended in a sequence of `sudo` commands over SSH, because
`UploadStateAndRestartStep` replaces the whole state directory, which deletes
the checkpoint the destination subnet halted at.

Deleting that checkpoint is what we want. It does not hold the canisters of the
subnet being merged away, so a replica coming up on it serves a state that
silently lost them -- a failure mode this test used to guard against by starting
the replica only once the recovery CUP existed. With the state directory
replaced, the merged state is the only state the destination subnet can resume
from, so the failure mode is gone rather than guarded against.

The upload step also restarts the replica, so the two proposals of the merge now
run before it rather than after, which is the order subnet splitting uses:
propose, upload, wait for the CUP, unhalt. Steps 14 and 15 swap accordingly.

The merged state is assembled in a directory of its own, as the upload step
insists that what it uploads hold a single checkpoint, and `R`'s states metadata
is copied next to it, as the step hands rsync every path it transfers as an
explicit source.

`UploadStateAndRestartStep` rather than
`Recovery::get_upload_state_and_restart_step`: the latter hardcodes the check
that the uploaded checkpoint matches the height an `ic-replay` run reported, and
this merge runs no `ic-replay`. Subnet splitting builds the step directly for
the same reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

An output nested under one of the input checkpoints was linked into itself: it
is created before the input is listed, so the linking descended into it over and
over, leaving a deeply nested partial checkpoint behind. Reject that, resolving
the paths first, as either side may reach the same directory through a link or a
`..`.

Remove the state sync checkpoint marker as well as the unverified one. A state
sync marker alone makes `checkpoint_status()` report `UnverifiedStateSync`, so
dropping only the unverified marker left an output the state manager would not
treat as verified, which was the point of dropping it.

Mark the files of the merged checkpoint read-only and sync them, as the state
manager does before a directory it assembled becomes a checkpoint. The files
linked in are read-only already, being the very files of the inputs, but the
marker written by the merge was not, and nothing reached the disk. Directories
stay writable, which is what a checkpoint's directories look like too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A relative output was rejected outright: resolving it walked off the front of
the path, since the ancestors of a bare relative path run out before reaching
the directory it is relative to. Make the path absolute before walking it. The
same oversight made the directory sync open the empty path, so that a merge with
a relative output failed after the rename had already gone through.

A merge that failed partway left its work at the output path, where a checkpoint
is expected: an incomplete one, named as a checkpoint but never marked, which
also blocked a retry through the check that the output must not exist. Assemble
in a staging directory next to the output instead and rename it into place once
the checkpoint is complete, removing it again if any step fails.

The staging directory is named so that it cannot be taken for a checkpoint,
whose name is a height in hexadecimal, and a leftover one is reported rather
than silently reused: a merge that was interrupted outright cannot clean up
after itself, and whoever does should know it was there.

The collision check now runs before anything is assembled, being the one failure
a caller is at all likely to hit, and compares the two inputs directly rather
than the output against the source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…own-test

# Conflicts:
#	rs/state_tool/src/commands/merge.rs

The markers are no longer removed from the merged checkpoint. Both inputs are
checkpoints a node had verified, which hold neither marker, so there was nothing
to remove; and removing one would have asserted that the result is a verified
checkpoint on the strength of nothing, where inheriting it says what is true.

The comments that explained more than the code does are gone with it, and the
test fixture writes the files a checkpoint really holds -- `canister.pbuf` in a
canister directory, `snapshot.pbuf` in a snapshot one -- through the constants
that name them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…own-test

Checking that the staging directory is free and then creating it are two steps,
so two merges of the same output could both get past the check and assemble into
the same directory, where the first to fail would remove what the other was
still putting together.

Create it instead, which is one step that only one of them can win, and report
the directory as left behind by an interrupted merge when the creation says it
is already there. Its parents are still created as before, so an output whose
directory does not exist yet keeps working.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…own-test

…ing-down-test

…ing-down-test

The `state-tool merge` command landed on master as #11469, so its files
conflicted with this branch's copy of the same work:

* `commands/merge.rs`: kept this branch's version, which carries the
  follow-ups made after #11469 was cut (assembling in a staging directory
  and renaming, claiming that directory by creating it, the nesting check
  for the output, and the trimmed marker handling), and took over the one
  refinement that is only on master: the tests now place snapshots at
  their real depth, `snapshots/<canister>/<snapshot>`, and assert down to
  the snapshot file.
* `main.rs`: kept this branch's `--output` documentation, as the nesting
  constraint master documents there is one `do_merge` now checks itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ing-down-test

Recording the in-progress ingress messages after a subnet merge landed on
master as #11365, in a version that was refined after this branch's copy
of that work was written: `after_merge()` is given the batch time to
timestamp the entries it records, destructures the state so that a new
field forces a decision, and re-partitions the input schedules through the
extracted `repartition_input_schedules()`. Master's version supersedes
this branch's throughout, so all four conflicting files (`after_merge()`
and its caller in `message_routing.rs`, plus both test files) are taken
from master.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… it is merged into

A subnet that is cooling down now also stops serving query calls (#11495),
so the two steps of this test that queried a canister of `M` while it was
cooling down can no longer do so:

* Step 10 checked that `U1`'s `install_code` calls installed the universal
  canister module, which a query reveals because a canister without a
  module rejects every query. That check moves to step 17, where `R`
  serves the merged-in canisters and answers for them again. Step 10 now
  checks the rejection itself, as the counterpart to step 8's ingress
  rejection.
* Step 11 read the iteration counters of both call loops to show they are
  stalled. `UT` is on `T` and is still read directly; for `US` on `M`, the
  number of rounds `M` skipped canister execution in stands in, which is
  the stronger statement: while it grows, `M` executes no canister message
  at all.

Verified with `bazel test //rs/tests/message_routing:subnet_cooling_down_test_head_nns`,
which passed in 1090s (the plain `subnet_cooling_down_test` target is
tagged `manual`: it runs the mainnet NNS, which knows neither the
`cooling_down` subnet record field nor the `MergeSubnets` proposal).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Drops this branch's follow-ups to `commands/merge.rs`, which were written
after #11469 was cut, so that the file is master's again: assembling in a
`<output>.merging` sibling and renaming on success (`staging_path()`,
`assemble()`), claiming that directory by creating it rather than checking
that it is free, and the `resolve()`-based check refusing an output nested
under either input, along with the three tests covering those. The
`--output` documentation in `main.rs` goes back to master's wording, which
states the nesting constraint that is once again the caller's to keep.

The subnet merge test is unaffected: it passes an output under its own
`recovery_merged` directory, outside both input checkpoints, and never
looks for a staging directory.

Verified with `bazel test //rs/tests/message_routing:subnet_cooling_down_test_head_nns`,
which passed in 1055s, assembling the merged state with the reverted
command.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Whether a subnet is halted is only observable in the replica's journal
today, as the line batch delivery logs every five seconds instead of
delivering a batch. Report it as `consensus_status{status=...}`, an
`IntGaugeVec` holding 1 for the status that batch delivery last computed
and 0 for the other two, so that the question "is this subnet halted right
now" can be asked of the metrics that already answer the rest of the
subnet merging dashboard's conditions.

The gauge is set where the status is computed, which is also the only place
that computes it, so it says what that path saw the last time it looked: a
subnet that stopped producing blocks altogether keeps reporting the status
that made it stop. `deliver_batches` takes the observer as a parameter next
to its `result_processor`, and `deliver_batches_for_ic_replay` passes a
no-op, leaving its signature unchanged.

The three statuses are reported separately rather than as one number
because they are not ordered, and the distinction matters: `Halting` still
produces empty blocks, so the consensus pool keeps moving, while `Halted`
produces none and the latest checkpoint is final. The log line fires for
both, so it cannot tell them apart.

Step 12 of the subnet merge test now waits for
`consensus_status{status="halted"}` on the node whose state it is about to
download, rather than opening an SSH session and searching that node's
journal for a log message. The height it halted at comes from
`state_manager_last_computed_manifest_height` instead of listing the
checkpoint directory over SSH, which is the stronger signal: a checkpoint
whose manifest is not computed yet has no CUP, and the manifest is what the
recovery proposal's state hash is compared against.

Verified with `bazel test //rs/tests/message_routing:subnet_cooling_down_test_head_nns`,
which passed in 1025s, reporting `M halted at checkpoint 2000, R halted at
checkpoint 2500` off the metrics and making no `journalctl` call at all,
plus the seven affected unit and integration tests of `//rs/consensus`,
`//rs/replay` and `//rs/state_machine_tests`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Step 12 of the subnet merge test waited until the node whose state it
downloads reported `consensus_status{status="halted"}` and then read the
height of the checkpoint it stopped in off
`state_manager_last_computed_manifest_height`. It now waits for the CUP the
subnet halts at instead, the way `subnet_splitting_test` does: poll
`_/catch_up_package` on that node until the registry version of the CUP's
summary is at least the one the `halt_at_cup_height` proposal created. That
version is the one `should_halt` reads the flag at, so such a CUP is the one
batch delivery stops at, and its height is the height of the last checkpoint
the subnet wrote.

The CUP is the better signal in both halves: it names the state the subnet
came to rest in, and it exists only once that state is certified and its hash
agreed upon, which is what the recovery proposal's state hash is compared
against. The two metrics together could not say that: a subnet that has just
stopped delivering batches may not have hashed the checkpoint it stopped at
yet, and the last computed manifest height is then the previous checkpoint, a
whole DKG interval before the state the merge is meant to be assembled from.
As in the reference, the node's certification height is checked against the
CUP height as well, so that a node holding a CUP the rest of the subnet
assembled before it got there is waited for, and one that ran past the CUP it
should have halted at fails the test.

That leaves `consensus_status` without a user, so the gauge, `observe_status`
and the `status_observer` parameter threaded through `deliver_batches` in
ef027ec are dropped again: `rs/consensus` is master's once more.

Verified with `bazel test //rs/tests/message_routing:subnet_cooling_down_test_head_nns`,
which passed in 1045s, reporting `M halted at checkpoint 2000, R halted at
checkpoint 2500`, the heights of the previous run, after waiting 250s on `M`
for the CUP at 2000 to replace the one at 1500, whose registry version still
preceded the halting one. Plus the seven affected unit and integration tests
of `//rs/consensus`, `//rs/replay` and `//rs/state_machine_tests`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ugh it

Adds `rs/recovery/subnet_merging`, the operator tool that performs a subnet
merge, built like `rs/recovery/subnet_splitting`: a resumable sequence of
steps, each of which either submits a proposal through `ic-admin`, waits for a
condition, or works on the states of the two subnets.

The steps: label the source subnet as "cooling down" and read back the registry
version `V` this created; wait until the subnet is ready to be merged, i.e.
until every term of the readiness condition of the `Subnet merging` dashboard
holds for `V` (all subnets at `V`, no message in any stream in either direction,
nothing but `processing` entries in the ingress history, empty subnet queues, no
subnet call context, and an empty refund pool); halt both subnets at their next
CUP and wait until the node each state is taken from holds that CUP -- served at
its public endpoint, written to its disk, and with the state it names certified;
stop both replicas, download both states and validate each against its CUP and
the subnet's public key in the NNS signed state tree; assemble the merged state
and compute the height, block time and state hash the recovery needs; reroute
the canister id ranges to the destination subnet, recover it at the merged state,
upload that state and wait for the recovery CUP; unhalt it; and, once every other
subnet routes the merged canisters to it, delete the subnet that was merged away.

The readiness condition is evaluated by the tool itself, by scraping the
metrics of every node of every subnet, in addition to printing the dashboard
link for the operator to confirm.

Shared code this needs:

* `ic-admin propose-to-merge-subnets`, as the `MergeSubnets` NNS function had
  no subcommand yet and every step of these tools proposes through `ic-admin`;
* an explicit block time for a recovery CUP proposal, as the merged state starts
  later than the checkpoints it was assembled from;
* the batch time of a checkpoint as a `state_tool` library function, rather than
  only as something the binary prints;
* a versioned subnet record getter on `RegistryHelper`, which polls the local
  store like every other getter: the halting CUP names a registry version that
  was created moments ago;
* `ic_cup_explorer::get_cup` and `ic_recovery::{get_member_node_ids_and_ips,
  steps::DownloadIcDataStep}` made public: a merge downloads two states, so it
  cannot use the helper that downloads into the single working directory of a
  recovery.

`subnet_cooling_down_test` now drives the tool instead of carrying its own
implementation of the merge: it sets the scenario up, hands over to the tool,
and makes its checks in between the tool's steps, at the points the runbook
names. Like the subnet splitting test, the part that drives the tool is plain
synchronous code -- the tool blocks on runtimes of its own -- with the
asynchronous checks driven through a runtime that lives for the whole test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…tools

`subnet_merging` was created as a sibling of `subnet_splitting` and took a
copy of the parts of it that are not specific to splitting a subnet. Both
tools halt a subnet at a CUP, work on the state it halted at, validate what
they produced against that CUP and the NNS signed state tree, and let an
operator confirm a dashboard before the next proposal -- so those parts now
live in one place, `rs/recovery/subnet_tools`, which both tools depend on:

* `agent_helper`: the NNS signed state tree and the subnet public key in it;
* `validation`: state tree -> subnet key -> CUP signature -> manifest root hash;
* `state_tool_helper`: the `state_tool` commands these tools run, i.e.
  computing and verifying a manifest, splitting a manifest, merging two
  checkpoints and reading the batch time of one;
* `utils`: reading a CUP, its block time, and the state hash of a checkpoint;
* `admin_helper`: halting a subnet at its next CUP height, and the arguments
  naming the subnets an operation moves canister id ranges between;
* `cli`: printing a dashboard URL and waiting for the operator's confirmation;
* `steps`: `ReadRegistryStep`, which is generic over what it reads.

What is specific to either operation stays where it was: the canister migration
proposals, the state splitting step and the expected manifest cross-check in
`subnet_splitting`, and the cooling down, merge and delete proposals, the merge
readiness evaluation, the halting CUP wait and the state merging step in
`subnet_merging`.

The dependencies the moved code brought with it leave the two tools, and
`state_tool_lib` is now visible to `//rs/recovery:__subpackages__` rather than
to two packages by name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>